Automated firmware settings verification

ABSTRACT

Systems and methods are described for managing computing resources. In one embodiment, data representative of an abstracted firmware framework is maintained. The data may comprise computing firmware settings and determined based on standardized associations between vendor-specific firmware settings and abstracted firmware settings that are independent of the vendor-specific firmware settings. In response to receiving a request for a computing firmware setting, the requested computing firmware setting is translated to one or more vendor-specific firmware settings based on the data. A computing resource capable of implementing the one or more vendor-specific firmware settings is identified.

BACKGROUND

A data center is a facility that houses computer systems and variousnetworking, storage, and other related components. Many organizationsand businesses operate and maintain data centers to provide computingand information services to support their day-to-day operations. Datacenters may also provide computing services on a permanent or anas-needed basis to businesses and individuals as a remote computingservice or to provide “platforms as a service” or “software as aservice” (e.g., cloud computing). The computing resources provided by adata center may include various types of resources, such as dataprocessing resources, data storage resources, data communicationresources, and the like.

To facilitate increased utilization of data center resources,virtualization technologies may allow a single physical computingmachine to host one or more instances of virtual machines (VMs) thatappear and operate as independent computer machines to a remotelyconnected computer user. With virtualization, the single physicalcomputing device can create, maintain, or delete virtual machines in adynamic manner. When a customer of a data center requests a new virtualmachine instance, the data center may provide a virtual machinemanagement service that identifies a “slot” for executing the newinstance. Customers may sometimes request changes to a virtual machineinstance or request a particular configuration. Some changes may requireupdates to firmware—a combination of software and hardware such as ahardware device with data stored in read-only memory.

BRIEF DESCRIPTION OF THE DRAWINGS

Throughout the drawings, reference numbers may be reused to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

FIG. 1 is a diagram illustrating a mechanism for providing a firmwaresettings framework in accordance with the present disclosure;

FIG. 2 illustrates an example computer system that may be used in someembodiments;

FIG. 3 is a diagram illustrating computing firmware groupings;

FIG. 4 is a diagram illustrating a mechanism for providing a firmwaresettings framework in accordance with the present disclosure;

FIG. 5 is a flowchart depicting an example procedure for providing afirmware settings framework in accordance with the present disclosure;

FIG. 6 is a flowchart depicting an example procedure for providing afirmware settings framework in accordance with the present disclosure;and

FIG. 7 is a flowchart depicting an example procedure for providing afirmware settings framework in accordance with the present disclosure.

DETAILED DESCRIPTION

The following detailed description is directed to technologies forproviding a service that implements one or more levels of abstractionfor updating various computer firmware settings. Many computer settingscan only be altered, for example, via firmware or Basic Input/OutputSystem (BIOS) updates that may be vendor specific. One issue that mayarise when altering such settings is during testing of new hardware orwhen operating hardware in a computing environment such as a datacenter. In the examples described herein, a data center is one exampleenvironment in which the described embodiments can be implemented.However, the described concepts can apply generally to other computingenvironments, for example across multiple data centers or locations.

Because of the limitations typically involved with modifying firmwaresettings, it can be difficult to automate the testing and verificationof settings that are controlled by firmware such as, for example,Non-Uniform Memory Access (NUMA), memory speed, power settings, and thelike. It can also be difficult to automate and manage the configurationof such firmware settings in a production capacity.

The present disclosure describes a firmware abstraction mechanism for aservice that provides one or more levels of abstraction for changingvarious firmware settings. Such a firmware abstraction mechanism caninclude the creation and management of workflows for firmware settingsby querying and changing firmware settings, mapping the settings tospecific hardware, changing and managing the settings in a controlledmanner, and removing/bringing affected devices back into service. Thefirmware abstraction mechanism can be used as part of a test workflowfor verifying performance and operation of different firmwareconfigurations. The firmware abstraction mechanism can also be used aspart of a production workflow to configure capacity at runtime so thatvarious system configurations with different firmware settings can beavailable for customers.

FIG. 1 is a diagram illustrating a computing environment 100 including amechanism for providing a firmware abstraction framework in accordancewith the present disclosure. In the present disclosure, a firmwareabstraction mechanism may also be referred to as a firmware abstractionframework or a firmware settings framework. The terms may be usedinterchangeably. Referring to FIG. 1, computing environment 100 mayinclude virtual machine instances 110 and 120 that may execute, forexample, on one or more server computers 130 and 140. It will beappreciated that some embodiments may involve additional virtual machineinstances that may be instantiated on additional server computers incomputing environment 100.

FIG. 1 also illustrates a public network 150 that may include one ormore computing devices such as computers 160 and 170. According to oneembodiment, virtual machine instance 110 may be configured to providecomputing services to a computer user (not shown) of public network 150via a gateway 190 and computers 160 and 170. For example, virtualmachine instance 110 may provide a set of remote access enterpriseapplications to a group of users who may, for example, be employees ofan enterprise customer.

A user, administrator, service or any computing resource in computingenvironment 100 may send a request to a firmware settings framework 180for a resource instance with a particular firmware setting. In oneembodiment, firmware settings framework 180 may maintain a record ofgroupings of computing resources that have been determined to be capableof meeting a predetermined baseline firmware configuration. Bymaintaining such groupings, computing firmware settings framework 180can efficiently identify and allocate computing resources for respondingto firmware setting requests. Such settings may be requested directly bya customer of the data center, by an administrator of the data center, aservice or any computing resource within the data center such as server130. Server 130 may also send a request on behalf of itself, or onbehalf of other servers.

In response to the request for firmware settings, firmware settingsframework 180 may access a list of available firmware baselineconfigurations. As used herein, firmware can be any combination ofsoftware and hardware, typically programs or data written in permanentstorage (e.g., read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), NAND or othernon-volatile storage on a device). The list of available firmwarebaseline configurations may be prioritized based on factors such as costand policy information. Firmware settings framework 180 may also accessinformation describing verification results and verification schedules.Firmware settings framework 180 may send information regarding theavailable firmware baseline configurations to the requestor.

In some embodiments, firmware settings framework 180 may receive arequest for one or more firmware capacity groupings based on a newfirmware setting. In response, firmware settings framework 180 maydetermine which, if any, of the plurality of available firmware capacitygroupings that may already be able to provide the new firmware setting.Firmware settings framework 180 may determine that such a groupingalready exists by reviewing the list of available firmware baselineconfigurations and requirements for verification of the new firmwaresetting. If such a grouping does not exist, then firmware settingsframework 180 may perform, or cause performance, of one or moreverification tasks to determine that the new firmware setting meetsestablished performance and capacity requirements. Once verification hasbeen performed, a new grouping can be formed consisting of resourcesthat have incorporated the new firmware setting. If such a groupingalready exists, then firmware settings framework 180 may sendinformation to the requestor regarding the existing grouping and themembers of the group.

In some embodiments, firmware settings framework 180 may sendinformation regarding requirements for verification of the new firmwaresetting to a resource such as server computer 130 if the new firmwaresetting is to be incorporated in the resource. The resource may performverification tasks and send the results of the verification tasks tofirmware settings framework 180. Firmware settings framework 180 maythen approve or disapprove the results and create a firmware capacitygrouping. As an example, a server computer may be configured to supportVM instances with a baseline configuration that includes a specifiedfirmware setting. The server computer will not be able to host VMinstances with a different firmware setting requirement unless theserver computer's firmware settings have been changed. Once the servercomputer has been changed and the update has been verified, firmwaresettings framework 180 can determine that the server computer can nowsupport VM instances with the updated setting requirement. Firmwaresettings framework 180 can then include the server computer in acapacity pool of server computers that can support such VM instances.

As described above, firmware is typically specific to vendorimplementation. One of the issues raised by settings that are firmwaredependent is the testing of computing configurations. To thoroughly testcomputing configurations, it is desirable to iterate test conditionsthrough the various possible firmware settings. Firmware settings mayrequire vendor specific procedures or necessitate the reboot of hardwareto a particular execution environment. After such changes, anotherreboot may be required. Some devices may provide a number of firmwarecustomization settings that may be accessed through various means. Someexamples of firmware controlled settings include the enabling of clockfrequency scaling (e.g., Intel Turbo), symmetric multithreading (e.g.,Intel hyper-threading) and NUMA settings.

In a data center environment, it is desirable to test computingconfigurations by iterating through various settings—including firmwaresettings—in a controlled way to determine optimal settings for aparticular workload or VM instance type. When new hardwareconfigurations are added in response to requests for unique firmwaresettings, fleet fragmentation and proliferation can result. By testingcomputing configurations, it is possible to achieve greater integrationof the computing configuration into the data center's products andservices. And by testing a variety of configurations, a given set ofhardware may be determined to be able to support a number of differentfirmware settings. Pools or groupings of computing resources can beidentified based on such capabilities and maintained based on frequentlyused settings and configurations.

In some cases a customer may request a particular setting that requiresa change to a firmware setting that is not currently provided by thedata center. A service in the data center may be implemented in one ormore computing devices. The service may be configured to determine asuitable computing device can potentially accommodate the setting,initiate a workflow to update and validate the particular setting, andreport that the update has been incorporated when the computing deviceis ready to provide the particular setting to the customer. Theparticular setting may also be made available to other customers who mayrequest similar settings.

In various embodiments, a firmware abstraction framework such asfirmware settings framework 180 of FIG. 1 may be implemented in one ormore computing devices and configured to receive requests for computingsettings and determine one or more firmware settings that willincorporate the requested computing setting. For example, the firmwareabstraction framework may map requests for computing settings to actualsettings that can be implemented in firmware settings (e.g., NUMA to“interleaved memory”).

The firmware abstraction framework may create workflows to update andvalidate specific settings on specific resources (i.e., servers). Thefirmware abstraction framework may identify one or more resources thatalready includes the requested setting or identify one or more settingsthat can be updated to provide the requested setting. For example, thefirmware abstraction framework may track pools of resources (i.e.,servers) that can support a given settings configuration. The firmwareabstraction framework may create also workflows to update and validatespecific settings on specific computing resources.

The firmware abstraction framework may also be configured to optimizethe placement for requested resources that have particular firmwaresettings requirements based on various factors such as minimization ofdisruption to existing services. The firmware framework may thus managedata center workflows to optimize the firmware updates and computingresource tracking capabilities.

Management of the firmware setting is also useful for managing devicesthat are associated with firmware updates such as flash storage devices.For example, flash devices have variations in write endurance and afirmware framework can track and manage the number of write/erase cycleson the devices and limit or throttle the number of times firmware isupdated on a particular device.

Firmware settings can be an important aspect of correct operation of theoperating system, and an incorrect or incompatible setting can disable acomputing resource or otherwise render it unusable. Furthermore,firmware settings have the potential to cause physical damage to devicesor to affect hardware reliability because the firmware settings can setcontrols for hardware such as clock rate and thermal throttling. Thefirmware abstraction framework can be configured to manage such hardwaresettings so that failure can be minimized. For example, the firmwareabstraction framework can determine how often to perform firmwareupdates on a given device and the conditions for testing the firmwaresettings. The firmware abstraction framework can also look for settingsthat may result in various blacklist behaviors that are to be avoided bythe managed devices.

As discussed above, firmware is often vendor specific, and in oneembodiment the firmware abstraction framework can implement anapplication programming interface (API) to provide an interface by whichvendor specific settings can be translated or “mapped” into a set ofabstracted settings that are not vendor specific. In this way, customersneed only identify the abstracted firmware settings and need not beconcerned with hardware specific settings that may vary across vendors.Such an API can implement interfaces for common denominators across adata center's computing resources in a way that provides commonality andlong-term compatibility for firmware settings regardless of the datacenter's hardware resources at any given time.

In some embodiments, the firmware abstraction framework may beconfigured to interact with a test framework that implements a mechanismfor tests and verification of assets in a data center. For example, whena request for a computing setting is received and it is determined thatthe computing setting requires firmware changes that have not beenpreviously verified, the firmware abstraction framework can identify aset of firmware settings that needs to be tested and verified to confirmthat the settings meet data center criteria. The firmware abstractionframework can also determine which of the settings are best suited tocomply with the requested computing setting. The firmware settings to betested can be sent to a test service to carry out the tests.

In some embodiments, the firmware abstraction framework may beconfigured to include an expert system and a knowledge base to provide adecision-making capability regarding the search and selection offirmware settings. The expert system can consider factors such as systemthroughput, processor utilization, and network bandwidth. Furthermore,the firmware abstraction framework may employ one or more fitnessfunctions to determine how close a given configuration is to achievingone or more system criteria. A configuration management mechanism may beused to perform permutation testing and determine optimal searchsettings. In one embodiment, a genetic algorithm may be used as a searchheuristic to efficiently determine searches for satisfactory firmwaresettings. In other embodiments, other search functions or combinationsof search functions can be used, such as a simulated annealing algorithmor a Hidden Markov Model algorithm.

In one example use case, a customer may request options for implementinga computing resource. In response, the firmware abstraction frameworkmay determine if the data center has the capacity to fulfill the requestand select a suitable computing device. The firmware abstractionframework may take the computing device out of service, update thefirmware and reboot the computing device. The firmware abstractionframework may then perform verification of the changes, and updateconfiguration information to track the new configuration.

The firmware abstraction framework can include a workflow managementcomponent that may be configured to select candidate computing devicesor resources and to move VM instances between computing devices asnecessary.

In some embodiments, the firmware abstraction framework can also includea billing component. In one embodiment, a pricing structure can bedetermined based on the settings selected by a customer. For example, abaseline price can be charged for standard computing configurations, anda premium price may be charged for special configurations that are notsupported by a standard resource pool or an existing resource andotherwise result in special provisioning to accommodate the request. Forexample, the dedication of a computing device with a uniqueconfiguration may result in an underutilization of the device,especially if additional VM instances cannot be hosted on the device.The premium price can include a set fee or an hourly premium or acombination of the two.

Thus in various embodiments the firmware abstraction framework may beused to update and manage firmware changes across the entire fleet ofcomputing resources in a data center.

Various aspects of the disclosure are now described with regard tocertain examples and embodiments, which are intended to illustrate butnot to limit the disclosure. It should be appreciated that the subjectmatter presented herein may be implemented as a computer process, acomputer-controlled apparatus, a computing system, or an article ofmanufacture, such as a computer-readable storage medium. While thesubject matter described herein is presented in the general context ofprogram modules that execute on one or more computing devices, thoseskilled in the art will recognize that other implementations may beperformed in combination with other types of program modules. Generally,program modules include routines, programs, components, data structures,and other types of structures that perform particular tasks or implementparticular abstract data types.

Those skilled in the art will also appreciate that the subject matterdescribed herein may be practiced on or in conjunction with othercomputer system configurations beyond those described herein, includingmultiprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, handheld computers,personal digital assistants, e-readers, cellular telephone devices,special-purposed hardware devices, network appliances, and the like. Theembodiments described herein may also be practiced in distributedcomputing environments, where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote memory storage devices.

In the following detailed description, references are made to theaccompanying drawings that form a part hereof, and that show, by way ofillustration, specific embodiments or examples. The drawings herein arenot drawn to scale. Like numerals represent like elements throughout theseveral figures.

FIG. 2 illustrates an example computing environment in which theembodiments described herein may be implemented. FIG. 2 is a diagramschematically illustrating an example of a data center 210 that canprovide computing resources to users 200 a and 200 b (which may bereferred herein singularly as “a user 200” or in the plural as “theusers 200”) via user computers 202 a and 202 b (which may be referredherein singularly as “a computer 202” or in the plural as “the computers202”) via a communications network 230. Data center 210 may, forexample, correspond to computing environment 100 in FIG. 1.

Data center 210 may be configured to provide computing resources forexecuting applications on a permanent or an as-needed basis. Thecomputing resources provided by data center 210 may include varioustypes of resources, such as data processing resources, data storageresources, data communication resources, and the like. Each type ofcomputing resource may be general-purpose or may be available in anumber of specific configurations. For example, data processingresources may be available as virtual machine instances. The instancesmay be configured to execute applications, including Web servers,application servers, media servers, database servers, and the like. Datastorage resources may include file storage devices, block storagedevices, and the like.

Each type or configuration of computing resource may be available indifferent sizes, such as large resources—consisting of many processorcores, large amounts of memory, and/or large storage capacity—and smallresources—consisting of fewer processor cores, smaller amounts ofmemory, and/or smaller storage capacity. Customers may choose toallocate a number of small processing resources as Web servers and/orone large processing resource as a database server, for example.

Data center 210 may include servers 216 (which may be referred hereinsingularly as “a server 216” or in the plural as “the servers 216”) thatprovide computing resources available as virtual machine instances 218(which may be referred herein singularly as “a virtual machine instance218” or in the plural as “the virtual machine instances 218”). Thevirtual machine instances 218 may be configured to execute applications,including Web servers, application servers, media servers, databaseservers, and the like. Other resources that may be provided include datastorage resources (not shown), and may include hard drives, solid statestorage drives or other storage devices, and the like.

The availability of virtualization technologies for computing hardwarehas provided benefits for providing large scale computing resources forcustomers and allowing computing resources to be efficiently andsecurely shared between multiple customers. For example, virtualizationtechnologies such as those provided by VMWare or other virtualizationsystems may allow a physical computing device to be shared amongmultiple users by providing each user with one or more virtual machineinstances hosted by the physical computing device. A virtual machineinstance may be a software emulation of a particular physical computingsystem that acts as a distinct logical computing system. Such a virtualmachine instance provides isolation among multiple operating systemssharing a given physical computing resource. Furthermore, somevirtualization technologies may provide virtual resources that span oneor more physical resources, such as a single virtual machine instancewith multiple virtual processors that spans multiple distinct physicalcomputing systems.

Referring to FIG. 2, communications network 230 may, for example, be apublicly accessible network of linked networks and possibly operated byvarious distinct parties, such as the Internet. In other embodiments,communications network 230 may be a private network, such as, forexample, a corporate or university network that is wholly or partiallyinaccessible to non-privileged users. In still other embodiments,communications network 230 may include one or more private networks withaccess to and/or from the Internet.

Communication network 230 may provide access to computers 202. Usercomputers 202 may be computers utilized by customers 200 or othercustomers of data center 210. For instance, user computer 202 a or 202 bmay be a server, a desktop or laptop personal computer, a tabletcomputer, a wireless telephone, a personal digital assistant (PDA), ane-book reader, a game console, a set-top box, or any other computingdevice capable of accessing data center 210. User computer 202 a or 202b may connect directly to the Internet (e.g., via a cable modem or aDigital Subscriber Line (DSL). Although only two user computers 202 aand 202 b are depicted, it should be appreciated that there may bemultiple user computers.

User computers 202 may also be utilized to configure aspects of thecomputing resources provided by data center 210. In this regard, datacenter 210 might provide a Web interface through which aspects of itsoperation may be configured through the use of a Web browser applicationprogram executing on user computer 202. Alternatively, a stand-aloneapplication program executing on user computer 202 might access anapplication programming interface (API) exposed by data center 210 forperforming the configuration operations. Other mechanisms forconfiguring the operation of the data center 210, including deployingupdates to an application, might also be utilized.

Servers 216 shown in FIG. 2 may be standard servers configuredappropriately for providing the computing resources described above andmay provide computing resources for executing one or more applications.In one embodiment, the computing resources may be virtual machineinstances 218. In the example of virtual machine instances, each of theservers 216 may be configured to execute an instance manager 220 a or220 b (which may be referred herein singularly as “an instance manager220” or in the plural as “the instance managers 220”) capable ofexecuting the virtual machine instances 218. The instance managers 220may be a virtual machine monitor (VMM) or another type of programconfigured to enable the execution of virtual machine instances 218 onserver 216, for example. As discussed above, each of the virtual machineinstances 218 may be configured to execute all or a portion of anapplication.

It should be appreciated that although the embodiments disclosed abovediscuss the context of virtual machine instances, other types ofimplementations can be utilized with the concepts and technologiesdisclosed herein. For example, the embodiments disclosed herein mightalso be utilized with computing systems that do not utilize virtualmachine instances.

In the example data center 210 shown in FIG. 2, a router 214 may beutilized to interconnect the servers 216 a and 216 b. Router 214 mayalso be connected to gateway 240, which is connected to communicationsnetwork 230. Router 214 may manage communications within networks indata center 210, for example by forwarding packets or other datacommunications as appropriate based on characteristics of suchcommunications (e.g., header information including source and/ordestination addresses, protocol identifiers, etc.) and/or thecharacteristics of the private network (e.g., routes based on networktopology, etc.). It will be appreciated that, for the sake ofsimplicity, various aspects of the computing systems and other devicesof this example are illustrated without showing certain conventionaldetails. Additional computing systems and other devices may beinterconnected in other embodiments and may be interconnected indifferent ways.

It should be appreciated that the network topology illustrated in FIG. 2has been greatly simplified and that many more networks and networkingdevices may be utilized to interconnect the various computing systemsdisclosed herein. These network topologies and devices should beapparent to those skilled in the art.

It should also be appreciated that data center 210 described in FIG. 2is merely illustrative and that other implementations might be utilized.Additionally, it should be appreciated that the functionality disclosedherein might be implemented in software, hardware, or a combination ofsoftware and hardware. Other implementations should be apparent to thoseskilled in the art. It should also be appreciated that a server,gateway, or other computing device may comprise any combination ofhardware or software that can interact and perform the described typesof functionality, including without limitation desktop or othercomputers, database servers, network storage devices and other networkdevices, PDAs, tablets, cellphones, wireless phones, pagers, electronicorganizers, Internet appliances, television-based systems (e.g., usingset top boxes and/or personal/digital video recorders), and variousother consumer products that include appropriate communicationcapabilities. In addition, the functionality provided by the illustratedmodules may in some embodiments be combined in fewer modules ordistributed in additional modules. Similarly, in some embodiments thefunctionality of some of the illustrated modules may not be providedand/or other additional functionality may be available.

The capacity of purchased computing resources provided by data center210 can be scaled in response to demand. In this regard, scaling refersto the process of instantiating (which may also be referred to herein as“launching” or “creating”) or terminating (which may also be referred toherein as “de-scaling”) instances of computing resources in response todemand. In this manner, the capacity of resources purchased by acustomer of data center 210 can be scaled on-demand.

Auto scaling is one mechanism for scaling computing resources inresponse to increases or lulls in demand for the resources. Auto scalingallows customers of data center 210 to configure data center 210 toscale their purchased computing resources according to conditionsdefined by the customer. For instance, rules may be defined for scalingup capacity in a particular manner in response to the occurrence ofspecified conditions, such as a spike in demand. Similarly, rules mightalso be defined to scale down capacity in a particular manner inresponse to the occurrence of other conditions, such as a lull indemand. The mechanisms disclosed herein for launching virtual machineinstances might be utilized when instances are manually launched by acustomer or when instances are launched by an auto scaling component indata center 210.

Data center 210 may also be configured with a deployment component toassist customers in the deployment of new instances of computingresources. The deployment component may receive a configuration from acustomer that includes data describing how new instances should beconfigured. For example, the configuration might specify one or moreapplications or software components that should be installed in newinstances, provide scripts and/or other types of code to be executed innew instances, and other types of information. The deployment componentutilizes the customer-provided configuration to launch and configurecustomer workloads on computing resources.

In computing environments such as those described herein, firmware isgenerally program code and data stored in persistent memory devices suchas ROM, EPROM, or flash memory. Changing the firmware of a device can bea fairly infrequent occurrence in most cases. Firmware such as the ROMBIOS of a computer typically configure functions of a device's hardware.Although firmware configuration has important ramifications for thesystem's operating system and thus the computer system's operation, mostsystems lack a mechanism for updating and managing firmware in anautomated and organized way. Firmware is typically updated manuallyusing a utility program, usually during the bootup sequence. Somefirmware in standalone devices rarely gets updated.

Computing devices, in particular in a data center scenario, may includea wide variety of hardware and software options that can be configuredby firmware. A data center must track and manage the configurations, andverify compatibility between the different configurations. In anembodiment, an automated firmware abstraction framework may include oneor more software layers for abstracting vendor specific firmwaresettings. In many computers, operating systems will run in conjunctionwith services provided by the system BIOS, which interfaces with thecomputing hardware. In various embodiments, an automated firmwareabstraction framework may include one or more software layers thatinteracts with the system BIOS and other firmware utility programs usinga standard interface. In an embodiment, the layers of the automatedfirmware abstraction framework provides an abstraction model to thesystems that run on it by providing a set of services and functions thatthe executing systems can use. In this way, the executing systems neednot deal with the specifics of the firmware but simply interact with theabstraction model.

By using the standard interfaces provided by the automated firmwareabstraction framework, it is possible to make changes to the underlyingcomputing devices and their firmware without updating software at higherlevels. In this way, compatibility can be maintained across servicesprovided in a data center while vendor specific changes areincorporated.

Some of the parameters that may be configured by an automated firmwareabstraction framework may include, but are not limited to, memoryinterleaving, processor clock frequency scaling, NIC parameters, harddrive parameters, access modes, ports, and the like.

FIG. 3 illustrates one embodiment in which computing resources aremanaged using a firmware settings framework. FIG. 3 includes firmwareconfiguration groupings 302, 304, 306, and 308 that each identify withone or more computing resources that meet or exceed the respectiverequirements for the respective firmware settings. In the figure,firmware configuration grouping 302 includes computing resources A, B,G, and H. Firmware configuration grouping 304 includes computingresources A, B, C, D, E, F, G, and H. Firmware configuration grouping306 includes computing resources A and B. Firmware configurationgrouping 308 includes computing resources G, H, M, N, P, and T. Each ofthe firmware configuration groupings 302, 304, 306, and 308 may beassociated with respective minimum firmware configurations 312, 314,316, and 318. As an example, firmware configuration 312 may requirecomputing resources with a specified firmware setting for a particularprocessor. In one embodiment, membership in the firmware configurationgroupings 302, 304, 306, and 308 may change as computing resources joinnew firmware configuration groupings or leave existing firmwareconfiguration groupings.

FIG. 4 illustrates an example computing environment in which theembodiments described herein may be implemented. Referring to thefigure, server computers 402, 410, 420, and 430 may communicate with afirmware settings framework 404 for access to computing firmwaregrouping information. Firmware settings framework 404 may in someembodiments correspond to firmware settings framework 180 as depicted inFIG. 1. Server computer 402 may host virtual machine instances 406 and408. Similarly, server computer 410 may host virtual machine instance412; and server computer 420 may host virtual machine instances 422,424, and 426. Server computer 430 may be configured to provide otherservices and may not be executing a virtual machine instance.

Server computer 402 may send a request for an updated computingconfiguration to firmware settings framework 404. Firmware settingsframework 404 may send information to server computer 402 indicatingapplicable computing firmware groupings, if any exist. If they do notexist, firmware settings framework 404 may identify requirements forestablishing a grouping of server computers that can support the updatedcomputing configuration. For example, requirements may include devicehardware and any software or firmware that needs to be installed, orexecution of a test to verify that a virtual computing instance of aparticular type can function with updated computing configuration. Theinformation may also indicate when evaluation tasks can be performed.For example, the information may include an evaluation schedule thatminimizes potential disruptions to existing services being provided tocustomers.

In one embodiment, the information describing the computing firmwaregroupings may be prioritized based on one or more criteria. For example,the computing firmware groupings may be prioritized based on costsassociated with providing the computing firmware groupings, or based onpolicies such as which computing firmware groupings have the highestdemand.

In some embodiments, the request for the updated computing configurationmay be sent to firmware settings framework 404 from server computer 402on behalf of one of the other server computers 410, 420, and 430. Inother embodiments, a third party such as a service executing on one ofthe server computers 402, 410, 420, and 430, or executing on anothercomputing device, may send the request on behalf of one or more of theserver computers 402, 410, 420, and 430.

Firmware settings framework 404 may cause the conduct of an evaluationand determine which, if any, of the requirements that can be met orexceeded by existing server computers. For example, server computer 402may be determined to be a candidate for incorporating the computingconfiguration and tasked with conducting an evaluation and determinewhich, if any, of the requirements that it can meet or exceed. Servercomputer 402 can optionally perform verification tasks that it canperform without jeopardizing its ability to continue uninterruptedhosting of its occupant VM instances since server computer 402 mayalready be a member of one or more computing firmware groupings and maycurrently host one or more virtual machines. Server computer 402 mayalso obtain additional details for verification from firmware settingsframework 404 or from some other source indicated by firmware settingsframework 180 in FIG. 1.

Server computer 402 may optionally send a request to join one or morecomputing firmware groupings to firmware settings framework 404. Servercomputer 402 may optionally include the cost of verifying servercomputer 402's ability to join each proposed computing firmwaregrouping. Firmware settings framework 404 may then evaluate the requestand determine whether to allow server computer 402 to proceed. Firmwaresettings framework 404 can make this determination using a number offactors. For example, firmware settings framework 404 may assess globalconsiderations such as the number of other server computers makingrequests and the number of available computing firmware groupings thatmay be rendered unavailable while server computer 402 as well as otherserver computers perform verification tests.

Firmware settings framework 404 may determine if the proposed computingfirmware groupings can accept additional members, if any potentialdisruptions to existing services are acceptable, and make otherdeterminations as necessary. Based on the determinations, firmwaresettings framework 404 may send an indication to proceed to servercomputer 402. Server computer 402, in response to receiving theindication to proceed, may then execute necessary verification tasks.The verification tasks may include, but are not limited to, runningsoftware tests, running VM instances or other workloads that simulatecustomer use cases, and gathering the results of the tests and usecases. Once the verification tasks have been completed, server computer402 may send the results to firmware settings framework 404 for reviewof the results and determination as to which computing firmwaregroupings that server computer 402 will be allowed to join.

Firmware settings framework 404 may analyze the information provided byserver computer 402 including the test results. Based on the receivedinformation and additional factors as necessary, the firmware settingsframework 404 may approve or disapprove admission to one or more of therequested computing firmware groupings. Firmware settings framework 404may, for example, consider admission approval/disapproval decisionsbased on the computing firmware groupings that still have room foradditional server computers, availability objectives for variouscomputing resources, and server administration policies. After sendingthe approval/disapproval information by firmware settings framework 404,server computer 402 may be designated as being associated with each ofthe approved computing firmware groupings. Firmware settings framework404 may optionally disassociate server computer 402 from some computingfirmware groupings. For example, firmware settings framework 404 mayhave implemented policies to remove server computers from lessvaluable/rare pools or overpopulated pools.

By using a predetermined set of baseline configurations and establishedtests for verifying compliance with the configurations, computingfirmware groupings can be efficiently maintained and newly addedfunctionality can be tracked by adding computing firmware groupings asneeded. Additionally, instead of taking server computers offline andtemporarily out of a computing firmware grouping to verify addedfunctionality, verification tests can be structured so that servercomputers can run the tests while they are hosting virtual services andwithout disrupting the hosted services.

In some embodiments, firmware settings framework 404 can use policiesand evaluation criteria to drive the computing firmware groupingpopulation to support certain computing resource management objectives.In one embodiment, computing firmware groupings can be assigneddifferent weights to influence requests submitted by server computers.For example, weights can be assigned so that computing firmwaregroupings are populated in a more cost effective manner according toadministrative policies or to evacuate server computers that have beenidentified for eventual removal from service. For instance, olderservers that are scheduled to be lease-returned can routinely be deniedpermission to join computing firmware groupings until the older serverseventually become unoccupied, at which point they can be lease-returned.

In some embodiments, computing firmware groupings can be managed so thatvarious availability objectives can be achieved. For example, weightscan be assigned to computing firmware groupings so that computingfirmware grouping availability can provide that at any point in time, anattempt to find a computing firmware grouping that providesfunctionality set X has a Y % chance of succeeding. Data for determiningthe values of X and Y can be based on a predetermined policy. Forexample, one such policy may be that a predetermined amount of reserveinstance capacity for a given set of attributes should be maintained.Other examples include ensuring that certain customer usage patterns canbe supported. For example, one such usage pattern can be for eachcapacity pool containing instances owned by entity Z, the computingfirmware groupings is managed such that an additional Q % of instancescan be accommodated. As another example, firmware settings framework 404can determine that the available servers in a certain computing firmwaregrouping are too low and should be increased. In other embodiments,historical data can be used to determine a computing firmware groupingmanagement policy.

Firmware settings framework 404 may reside on one or more servercomputers or other computing resources in a data center. Firmwaresettings framework 404 may in some embodiments be managed by a VMM orother management software executing in the data center. Firmwaresettings framework 404 may also execute on one or more virtual machines.

FIG. 5 illustrates an example operational procedure for managingcomputing resources in a data center using a firmware settingsframework. In an embodiment, a firmware settings framework cancorrespond to firmware settings framework 404 in FIG. 4 or firmwaresettings framework 180 in FIG. 1.

Referring to FIG. 5, operation 500 begins the operational procedure.Operation 500 may be followed by operation 502. Operation 502illustrates receiving a request for a computing resource with a desiredcomputing attribute. In an embodiment, the request may be received in acomputing environment comprising a plurality of computing devicesproviding computing resources.

Operation 502 may be followed by operation 504. Operation 504illustrates identifying at least one of the plurality of computingdevices to provide the hardware-specific firmware settings. Therequested computing resource may be deployed on the identified computingdevice.

Operation 504 may be followed by operation 506. Operation 506illustrates translating the requested computing attribute tocorresponding hardware-specific firmware settings on the identifiedcomputing device. In some embodiments, the translating may be performedbased on predetermined relationships between hardware-specific firmwaresettings and abstracted firmware settings that correspond to thehardware-specific firmware settings.

Operation 506 may be followed by operation 508. If the firmware settingsare not implemented on the identified computing device, then operation508 may be followed by operation 510. Operation 510 illustratesimplementing and verifying the hardware-specific firmware settings onthe identified computing device. For example, the hardware-specificfirmware settings can be verified on the identified computing device andthe requested computing resource can be provided on the identifiedcomputing device. However, it is possible that the identified computingdevice with the hardware-specific firmware settings does not meetapplicable requirements and the requestor may be informed that thedesired computing attribute cannot be provided. In some embodiments,additional candidate computing devices may be identified and verified toselect a device that can satisfy the applicable requirements.

If the firmware settings are implemented on the identified computingdevice, then operation 508 may be followed by operation 512. Operation512 illustrates providing the requested computing resource on theidentified computing device. Operation 512 may be followed by operation502.

FIG. 6 illustrates an example operational procedure for managingcomputing resources in a data center using a firmware settingsframework. In an embodiment, a firmware settings framework cancorrespond to firmware settings framework 404 in FIG. 4 or firmwaresettings framework 180 in FIG. 1.

Referring to FIG. 6, operation 600 begins the operational procedure.Operation 600 may be followed by operation 602. Operation 602illustrates maintaining mappings between a plurality of computingsettings of a plurality of computing devices in a computing environmentand corresponding firmware settings of the one or more computing devicesin the computing environment. In one embodiment, the mappings may berepresentative of a relationship between hardware-specific firmwaresettings and abstracted firmware settings that are independent of thehardware-specific firmware settings. For example, the abstractedfirmware settings may provide at least a degree of independence fromspecific hardware implementations by providing stable abstractedparameters that can be translated to a hardware-specific firmwaresettings without having to consider the details of the hardware-specificfirmware settings.

Operation 602 may be followed by operation 604. Operation 604illustrates receiving a request for a computing attribute related to oneof the plurality of computing settings.

Operation 604 may be followed by operation 606. If it is determined thatnone of the mappings corresponds to the requested computing attribute,then operation 606 may be followed by operation 608. Operation 608illustrates initiating a remediation process. In one embodiment, theremediation process can include causing execution of a process to createand verify a new mapping between the computing setting and one or morecorresponding firmware settings. For example, the computing setting canbe verified on a set of computing resources with a correspondingfirmware setting and a new mapping can be added. However, it is possiblethat the requested updated configuration does not meet applicablerequirements and the mapping can indicate that the requested updatedconfiguration is not valid.

If a mapping does exist, then operation 606 may be followed by operation610. Operation 610 illustrates selecting one of the mappings to providethe requested computing attribute. Operation 610 may be followed byoperation 602.

FIG. 7 illustrates an example operational procedure for managingcomputing resources in a data center using a firmware settingsframework. In an embodiment, a firmware settings framework cancorrespond to firmware settings framework 404 in FIG. 4 or firmwaresettings framework 180 in FIG. 1.

Referring to FIG. 7, operation 700 begins the operational procedure.Operation 700 may be followed by operation 702. Operation 702illustrates forming groupings of computing devices in a computingenvironment comprising a plurality of computing devices providingcomputing resources. In one embodiment, the groupings may have commoncomputing attributes corresponding to one or more firmware settings ofan abstraction firmware framework. Additionally, the abstractionfirmware framework may represent relationships between hardware-specificfirmware settings and abstracted firmware settings that are independentof the hardware-specific firmware settings.

Operation 702 may be followed by operation 704. Operation 704illustrates receiving a request for a computing attribute. Operation 704may be followed by operation 706. Operation 706 illustrates identifyingat least one of the firmware settings that correspond to the requestedcomputing attribute.

Operation 706 may be followed by operation 708. If it is determined thatnone of the groupings can support the identified at least one firmwaresetting, then operation 708 may be followed by operation 710. Operation708 illustrates creating a new grouping when it is determined that noneof the groupings can support the identified at least one firmwaresetting.

If a grouping can support the identified at least one firmware setting,then operation 708 may be followed by operation 712. Operation 712illustrates providing the requested computing attribute. For example,one of the computing devices in the grouping that can support thefirmware setting can be selected to provide the requested computingattribute. Operation 712 may be followed by operation 704.

Each of the processes, methods, and algorithms described in thepreceding sections may be embodied in, and fully or partially automatedby, code modules executed by one or more computers or computerprocessors. The code modules may be stored on any type of non-transitorycomputer-readable medium or computer storage device, such as harddrives, solid state memory, optical disc, and/or the like. The processesand algorithms may be implemented partially or wholly inapplication-specific circuitry. The results of the disclosed processesand process steps may be stored, persistently or otherwise, in any typeof non-transitory computer storage such as, e.g., volatile ornon-volatile storage.

The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure. In addition, certain method or processblocks may be omitted in some implementations. The methods and processesdescribed herein are also not limited to any particular sequence, andthe blocks or states relating thereto can be performed in othersequences that are appropriate. For example, described blocks or statesmay be performed in an order other than that specifically disclosed, ormultiple blocks or states may be combined in a single block or state.The example blocks or states may be performed in serial, in parallel, orin some other manner. Blocks or states may be added to or removed fromthe disclosed example embodiments. The example systems and componentsdescribed herein may be configured differently than described. Forexample, elements may be added to, removed from, or rearranged comparedto the disclosed example embodiments.

It will also be appreciated that various items are illustrated as beingstored in memory or on storage while being used, and that these items orportions of thereof may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the software modulesand/or systems may execute in memory on another device and communicatewith the illustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other ways, such as at leastpartially in firmware and/or hardware, including, but not limited to,one or more application-specific integrated circuits (ASICs), standardintegrated circuits, controllers (e.g., by executing appropriateinstructions, and including microcontrollers and/or embeddedcontrollers), field-programmable gate arrays (FPGAs), complexprogrammable logic devices (CPLDs), etc. Some or all of the modules,systems and data structures may also be stored (e.g., as softwareinstructions or structured data) on a computer-readable medium, such asa hard disk, a memory, a network, or a portable media article to be readby an appropriate drive or via an appropriate connection. The systems,modules and data structures may also be transmitted as generated datasignals (e.g., as part of a carrier wave or other analog or digitalpropagated signal) on a variety of computer-readable transmission media,including wireless-based and wired/cable-based media, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, the present invention may be practiced withother computer system configurations.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements, and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list.

While certain example embodiments have been described, these embodimentshave been presented by way of example only, and are not intended tolimit the scope of the inventions disclosed herein. Thus, nothing in theforegoing description is intended to imply that any particular feature,characteristic, step, module, or block is necessary or indispensable.Indeed, the novel methods and systems described herein may be embodiedin a variety of other forms; furthermore, various omissions,substitutions and changes in the form of the methods and systemsdescribed herein may be made without departing from the spirit of theinventions disclosed herein. The accompanying claims and theirequivalents are intended to cover such forms or modifications as wouldfall within the scope and spirit of certain of the inventions disclosedherein.

What is claimed is:
 1. A computing system comprising at least onecomputing node and at least one data store in communication with the atleast one computing node, the at least one data store having storedtherein computer instructions, that upon execution by the at least onecomputing node, cause at least: forming groupings of computing devicesin a computing environment providing computing resources, the groupingshaving common computing attributes corresponding to one or more firmwaresettings of an abstraction firmware framework, the abstraction firmwareframework having an associated interface for mapping relationshipsbetween hardware-specific firmware settings for a plurality of thecomputing devices and a plurality of selectable abstracted firmwaresettings that are translatable to the hardware-specific firmwaresettings, wherein the abstracted firmware settings are different fromthe hardware-specific firmware settings, wherein the abstracted firmwaresettings are not vendor-specific, and the abstracted firmware settingscorrespond to one or more desired computing attributes of the pluralityof computing devices; receiving a request for a selected abstractedfirmware setting; determining which of the groupings can support theselected abstracted firmware setting; and creating a new grouping whenit is determined that none of the groupings can support the selectedabstracted firmware setting, wherein the new grouping is based in parton verification that a computing device in the new grouping that hasimplemented the selected abstracted firmware setting meets or exceeds atleast one performance criterion.
 2. The computing system according toclaim 1, wherein the groupings are based on at least one computingenvironment management criterion.
 3. The computing system according toclaim 2, wherein the at least one computing environment managementcriterion is a firmware grouping policy.
 4. The computing systemaccording to claim 1, further comprising implementing an expert systemconfigured to provide a decision-making capability for forming thegroupings.
 5. A computer-implemented method for managing computingresources, comprising: maintaining data representative of an abstractedfirmware framework comprising computing firmware settings, the datadetermined based on standardized associations performed using aninterface associated with the abstracted firmware framework, theassociations between hardware-specific firmware settings of a pluralityof the computing resources and a plurality of abstracted firmwaresettings that are independent of the hardware-specific firmwaresettings, wherein the abstracted firmware settings are different fromthe hardware-specific firmware settings, wherein the abstracted firmwaresettings are independent of the hardware-specific settings, and whereinthe abstracted firmware settings correspond to one or more desiredcomputing attributes; verifying at least one of the hardware-specificfirmware settings to confirm that it meets predetermined criteria;receiving a request for a selected abstract firmware setting;translating the requested selected abstract firmware setting to one ormore vendor-specific firmware settings based on the data; andidentifying a computing resource capable of implementing the one or morevendor-specific firmware settings.
 6. The method of claim 5 wherein thehardware specific firmware settings include at least one of a basicinput/output system (BIOS), a non-uniform memory access (NUMA),processor clock rate, processor clock frequency scaling, performancestate or power state setting.
 7. The method of claim 5 furthercomprising maintaining groupings of computing resources, each of thegroupings comprising one or more computing resources that have a commonset of computing firmware settings that have been incorporated andverified.
 8. The method of claim 5 wherein the predetermined criteriaincludes a data center capacity management policy.
 9. The method ofclaim 7, further comprising sending an indication to a source of therequest as to which, if any, of the groupings have incorporated andverified the computing firmware settings.
 10. The method of claim 8wherein the data center capacity management policy is a targetavailability goal.
 11. The method of claim 5 further comprising causingidentification of at least one computing resource to incorporate andverify the selected abstract firmware setting and initiating at leastone verification task to verify that the computing resource satisfiesthe predetermined criteria after incorporating the selected abstractfirmware setting.
 12. The method of claim 11 further comprisingevaluating data received from the at least one computing resource andverifying that the computing resource satisfies the predeterminedcriteria after incorporating the selected abstract firmware setting. 13.The method of claim 5, further comprising using a fitness function todetermine whether the at least one of the hardware-specific firmwaresettings meets the predetermined criteria.
 14. One or morenon-transitory computer-readable storage media having collectivelystored thereon executable instructions that, when executed by one ormore processors of a computer system, cause the computer system to:receive a selected abstract firmware setting; and identify at least onecomputing resource capable of implementing firmware settingscorresponding to the selected abstract firmware setting, the at leastone computing resource identified based on a determination that one ormore hardware-specific firmware settings incorporated on the at leastone computing resource is compatible with the selected abstract firmwaresetting, wherein it is verified that the at least one computing resourcesatisfies predetermined criteria after implementation of the firmwaresettings, and wherein: a plurality of abstract firmware settings aremapped using an interface of an abstraction firmware framework to aplurality of hardware-specific firmware settings for a plurality ofcomputing resources; the abstract firmware settings are different fromthe hardware-specific firmware settings, and the abstract firmwaresettings are independent of vendor-specific settings.
 15. Thecomputer-readable storage media of claim 14 further storing thereonexecutable instructions that, when executed by the one or moreprocessors of a computer system, cause the computer system to adjust thefirmware settings based on a fitness function.
 16. The computer-readablestorage media of claim 15 further storing thereon executableinstructions that, when executed by the one or more processors of acomputer system, cause the computer system to use at least one criterionas a performance goal.
 17. The computer-readable storage media of claim14 wherein performance testing is performed iteratively to identify afinal set of firmware settings based on a fitness function.